Imaging apparatus, image processing apparatus, imaging system, imaging method, image processing method, and recording medium

ABSTRACT

An imaging apparatus includes an imaging device, a first imaging optical system and a second imaging optical system that form respective input images from mutually different viewpoints onto the imaging device, and a first modulation mask and a second modulation mask that modulate the input images formed by the first imaging optical system and the second imaging optical system. The imaging device captures a superposed image composed of the two input images that have been formed by the first imaging optical system and the second imaging optical system, modulated by the first modulation mask and the second modulation mask, and optically superposed on each other, and the first modulation mask and the second modulation mask have mutually different optical transmittance distribution characteristics.

BACKGROUND 1. Technical Field

The present disclosure relates to an imaging apparatus, an imagingmethod, and so forth for capturing an image.

2. Description of the Related Art

In systems such as driving safety support systems for automobiles,automatic control systems for mobile robots, or surveillance camerasystems for detecting a suspicious person or the like, the systems andthe users thereof need three-dimensional positional information of thesurroundings of the systems in order to made determinations or controlthe systems.

So-called binocular stereoscopic vision (also referred to astrigonometry) is widely used as a method of acquiring three-dimensionalpositions (see, for example, Japanese Unexamined Patent ApplicationPublication No. 6-167564). In the binocular stereoscopic vision, twocameras are arranged at mutually different viewpoints in such a mannerthat their fields of view overlap each other, and these cameras eachcapture an image. Then, a corresponding point between the two capturedimages is identified, and the three-dimensional position of thecorresponding point is calculated by using the identified correspondingpoint and information on the two cameras such as their positions andorientations obtained in advance.

The imaging apparatus disclosed in Japanese Unexamined PatentApplication Publication No. 6-167564, however, has a problem that itscircuit scale is large. In other words, since the imaging apparatusrequires the same number of imaging device (i.e., the aforementionedcamera) as the number of the viewpoints in order to capture images frommutually different viewpoints, there is a problem that the circuit sizeis large.

SUMMARY

One non-limiting and exemplary embodiment provides an imaging apparatusand so forth that can reduce the circuit size.

In one general aspect, the techniques disclosed here feature an imagingapparatus, and the imaging apparatus includes an imaging device, two ormore imaging optical systems that form respective input images frommutually different viewpoints onto the imaging device, and two or moremodulation masks that modulate the input images formed by the respectivetwo or more imaging optical systems. The imaging device captures asuperposed image composed of the two or more input images that have beenformed by the two or more imaging optical systems, modulated by the twoor more modulation masks, and optically superposed on each other, andthe two or more modulation masks have mutually different opticaltransmittance distribution characteristics.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable storage medium, or any selectivecombination thereof. Computer-readable storage media include, forexample, a nonvolatile storage medium, such as a Compact Disc-Read OnlyMemory (CD-ROM).

According to the present disclosure, the circuit size can be reduced.Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of animaging system according to a first embodiment;

FIG. 2 is a configuration diagram illustrating a specific configurationof the imaging system according to the first embodiment;

FIG. 3 is a schematic diagram illustrating another exemplaryconfiguration of an optical unit according to the first embodiment;

FIG. 4 illustrates an ideal optical system;

FIG. 5 illustrates an exemplary arrangement of an imaging apparatus andan exemplary imaging condition according to the first embodiment;

FIG. 6 illustrates input images of computer graphics, an observationimage based on the input images, and reconstructed images generated fromthe observation image according to the first embodiment;

FIG. 7 illustrates the distance calculated by a distance calculatingunit according to the first embodiment and the true value of thedistance for comparison;

FIG. 8 illustrates a configuration of two imaging optical systems in anoptical unit according to a modification of the first embodiment;

FIG. 9 illustrates another configuration of the two imaging opticalsystems in the optical unit according to the modification of the firstembodiment;

FIG. 10 illustrates an example of a simulation result according to asecond embodiment;

FIG. 11 illustrates an example of a simulation result according to thesecond embodiment;

FIG. 12 is a block diagram illustrating a configuration of an imageprocessing apparatus configured as a computer according to the presentdisclosure;

FIG. 13A is a flowchart illustrating an imaging method according to anaspect of the present disclosure;

FIG. 13B is a flowchart illustrating an image processing methodaccording to an aspect of the present disclosure; and

FIG. 13C is a flowchart illustrating an image processing methodaccording to another aspect of the present disclosure.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

The present inventor has found that the following problem arises in theimaging apparatus disclosed in Japanese Unexamined Patent ApplicationPublication No. 6-167564 described in the section titled “Description ofthe Related Art.”

In the binocular stereoscopic vision, the parallax, which is thedifference between the directions from two viewpoints, becomes 0 orapproaches 0 on or in the vicinity of the straight line connecting theviewpoints of two cameras, which thus leads to a problem in that thethree-dimensional position cannot be calculated. In particular, whencameras with a viewing angle of 180 degrees or greater are to be used,an area in which the parallax becomes 0 and the three-dimensionalposition thus cannot be calculated is inevitably included within thefield of view. Thus, the imaging apparatus disclosed in JapaneseUnexamined Patent Application Publication No. 6-167564 employs thefollowing method in order to acquire the three-dimensional position in awider viewing angle. Specifically, three or more cameras are used, andan area in which the three-dimensional position cannot be calculatedwith two of these cameras is interpolated with the three-dimensionalposition calculated with other two of these cameras. In addition,Japanese Unexamined Patent Application Publication No. 6-167564discloses an apparatus that includes cameras having fisheye lenses witha wide viewing angle with the purpose of following an object moving athigh speed within the field of view or following a plurality of objectssimultaneously. In this technique, fisheye images of a wide viewingangle are acquired with the cameras having the fisheye lenses, a movingobject is detected from each of the fisheye images, and a linearequation that passes through the detected moving object is calculated.Then, linear equation sets each composed of a set of a plurality oflinear equations pertaining to each moving object are obtained, and thusthe three-dimensional position is determined. The use of three or morecameras makes it possible to cover an area in which thethree-dimensional position cannot be calculated with given two of thesecameras by other cameras, and thus an area in which thethree-dimensional position cannot be calculated does not arise.

However, the imaging apparatus disclosed in Japanese Unexamined PatentApplication Publication No. 6-167564 has a problem that the circuit sizeis large, because it requires the same number of imaging device as thenumber of viewpoints in order to capture images from mutually differentviewpoints.

To address such a problem, an imaging apparatus according to an aspectof the present disclosure includes an imaging device, two or moreimaging optical systems that form respective input images from mutuallydifferent viewpoints onto the imaging device, and two or more modulationmasks that modulate the input images formed by the respective two ormore imaging optical systems, and the imaging device captures asuperposed image composed of the two or more input images that have beenformed by the two or more imaging optical systems, modulated by the twoor more modulation masks, and optically superposed on each other.

With this configuration, the two or more input images from mutuallydifferent viewpoints are optically superposed on each other and capturedby the single imaging device, and at this point, these input images havebeen modulated by the two or more modulation masks. Therefore, bysplitting the superposed image composed of the two or more input imagesthat have been modulated and superposed on each other with the use ofmodulation information that indicates a mode of the modulation masks,reconstructed images corresponding to the original two or more inputimages can be generated. As a result, the number of imaging devicesrequired in an imaging apparatus in order to capture a plurality ofinput images from mutually different viewpoints can be reduced to anumber smaller than the number of the viewpoints. In other words, aplurality of input images from mutually different viewpoints can beacquired simultaneously with a single imaging device. With thisconfiguration, the circuit size of the imaging apparatus can be reduced.

Here, the two or more modulation masks may have mutually differentoptical transmittance distribution characteristics. Specifically, thecorrelation coefficient of the optical transmittance distributioncharacteristics of the two or more modulation masks may be less than 1.To be more specific, the correlation coefficient of the opticaltransmittance distribution characteristics of the two or more modulationmasks may be substantially 0.

With this configuration, the two or more reconstructed images generatedfrom the superposed image can be made more closer to the original two ormore input images that have not been modulated and superposed on eachother.

In addition, the optical transmittances at respective sites in each ofthe two or more modulation masks may be in uniformly random numbers.Alternatively, the optical transmittances at respective sites in each ofthe two or more modulation masks may be in Gaussian random numbers.

With this configuration, the two or more reconstructed images generatedfrom the superposed image can be made appropriately closer to theoriginal two or more input images that have not been modulated andsuperposed on each other.

In addition, the imaging apparatus may include a plurality imaging setseach including the two or more imaging optical systems, the two or moremodulation masks, and the imaging device.

With this configuration, input images from a larger number of viewpointscan be captured.

An image processing apparatus according to an aspect of the presentdisclosure includes an acquirer that acquires a superposed imagecomposed of two or more input images from mutually different viewpointsthat have been modulated and optically superposed on each other, and animage processor that generates two or more reconstructed images bysplitting the superposed image with the use of modulation informationthat indicates a mode of modulation of the two or more input images.

With this configuration, the reconstructed images corresponding to theoriginal two or more input images can be generated from the superposedimage. As a result, the number of imaging devices required in an imagingapparatus in order to capture a plurality of input images from mutuallydifferent viewpoints can be reduced to a number smaller than the numberof the viewpoints. Thus, the circuit size of the imaging apparatus canbe reduced.

In addition, when generating the two or more reconstructed images, theimage processor may calculate a parallax between the two or morereconstructed images along with the two or more reconstructed images onthe basis of an evaluation value and may calculate the distance to anobject in the two or more reconstructed images on the basis of thecalculated parallax.

With this configuration, the distance to the object can be calculated,and thus the three-dimensional position around the mutually differentviewpoints can be calculated appropriately.

In addition, when generating the two or more reconstructed images, theimage processor may calculate a value of a first parameter correspondingto the two or more reconstructed images and a value of a secondparameter corresponding to the parallax that minimize the evaluationvalue that is based on the first parameter and the second parameter, mayconvert the calculated value of the first parameter to the two or morereconstructed images, and may acquire the calculated value of the secondparameter as the parallax. For example, the evaluation value may be asum of respective values indicated by a first term, a second term, and athird term; the first term may indicate, by using the first parameter, asum of squares of a difference between the superposed image and imagesobtained by modulating the two or more reconstructed images inaccordance with the modulation information; the second term may indicatea value obtained by weighting an L1-norm of the first parameter; and thethird term may indicate, by using the first parameter and the secondparameter, a value obtained by weighting an L1-norm of a differencebetween an image obtained by translating one of the two or morereconstructed images by a distance corresponding to the parallax andanother one of the two or more reconstructed images. The first parameterand the second parameter are, for example, c and D, which will bedescribed later, and the evaluation value is a value obtained through afunction following argmin in the expression (9), which will be describedlater.

With this configuration, the two or more reconstructed images and theparallax can be obtained from the evaluation value that is based on thefirst parameter and the second parameter, and the accuracy of thesereconstructed images and the parallax can be increased.

An image processing apparatus according to an aspect of the presentdisclosure includes an acquirer that acquires a superposed imagecomposed of two or more input images from mutually different viewpointsthat have been modulated and optically superposed on each other, and animage processor that calculates a parallax between the two or more inputimages by using modulation information that indicates a mode ofmodulation of the two or more input images and the superposed image andcalculates the distance to an object in the two or more input images onthe basis of the calculated parallax.

With this configuration, the distance to the object can be calculated,and thus the three-dimensional position around the mutually differentviewpoints can be calculated appropriately.

It is to be noted that general or specific embodiments of the above maybe implemented in the form of a system, a method, an integrated circuit,a computer program, or a computer-readable recording medium such as aCD-ROM, or through any desired combination of a system, a method, anintegrated circuit, a computer program, and a recording medium.

Hereinafter, embodiments will be described in concrete terms withreference to the drawings.

It is to be noted that the embodiments described hereinafter merelyillustrate general or specific examples. The numerical values, theshapes, the materials, the constituent elements, the arrangement andpositions of the constituent elements, the connection modes of theconstituent elements, the steps, the order of the steps, and so forthindicated in the embodiments hereinafter are examples and are notintended to limit the present disclosure. In addition, among theconstituent elements described in the embodiments hereinafter, aconstituent element that is not described in an independent claimindicating the broadest concept is described as an optional constituentelement.

Furthermore, the drawings are schematic diagrams and do not necessarilyprovide the exact depiction. In addition, constituent elements that areidentical across the drawings are given identical reference characters.

First Embodiment 1. Schematic Configuration of Imaging System

FIG. 1 illustrates a schematic configuration of an imaging systemaccording to a first embodiment.

As illustrated in FIG. 1, an imaging system 1 according to the firstembodiment includes an imaging apparatus 10 and an image processingapparatus 20. The imaging apparatus 10 captures an image in which twoimages from mutually different viewpoints are superposed on each other,and the image processing apparatus 20 processes the two images that aresuperposed on each other.

The imaging apparatus 10 includes two imaging optical systems 11L and11R, two modulation masks 12L and 12R, and an imaging device 13.

The two imaging optical systems 11L and 11R are optical systemsincluding a lens or the like for forming input images from mutuallydifferent viewpoints onto the imaging device 13.

The two modulation masks 12L and 12R modulate the input images formed bythe two imaging optical systems 11L and 11R, respectively. In each ofthe modulation masks 12L and 12R, the optical transmittance varies atdifferent sites thereof.

The imaging device 13 captures a superposed image composed of the twoinput images that have been formed by the two imaging optical systems11L and 11R, modulated by the two modulation masks 12L and 12R, andoptically superposed on each other.

The image processing apparatus 20 includes an acquiring unit 21 and animage processing unit 22.

The acquiring unit 21 acquires the superposed image composed of the twoinput images from mutually different viewpoints that have been modulatedand optically superposed on each other. Here, the superposed image isthe image captured by the imaging device 13 of the imaging apparatus 10.

The image processing unit 22 generates reconstructed imagescorresponding to the two input images by splitting the superposed imagewith the use of modulation information that indicates a mode ofmodulation of the two input images. In addition, the image processingunit 22 calculates the parallax between the two reconstructed images byusing the modulation information and the superposed image and calculatesthe distance to an object in the two reconstructed images on the basisof the calculated parallax.

Although the imaging apparatus 10 includes a single imaging set thatincludes the two imaging optical systems 11L and 11R, the two modulationmasks 12L and 12R, and the imaging device 13 in the first embodiment,the imaging apparatus 10 may include a plurality of such imaging sets.In this case, the image processing apparatus 20 acquires a plurality ofsuperposed images captured by the respective imaging sets and generatestwo reconstructed images for each of the plurality of superposed imagesby splitting each of the superposed images.

In addition, although the imaging apparatus 10 includes the two imagingoptical systems 11L and 11R and the two modulation masks 12L and 12R,the number of the imaging optical systems and the number of themodulation masks are each not limited to two and merely need to be atleast two.

When the imaging apparatus 10 includes two or more imaging opticalsystems and two or more modulation masks, the imaging device 13 capturesa superposed image composed of two or more input images that have beenformed by the two or more imaging optical systems, modulated by the twoor more modulation masks, and optically superposed on each other. Inthis case, the acquiring unit 21 of the image processing apparatus 20acquires the superposed image composed of the two or more input imagesfrom mutually different viewpoints that have been modulated andoptically superposed on each other. The image processing unit 22generates reconstructed images corresponding to the two or more inputimages by splitting the superposed image with the use of modulationinformation that indicates a mode of modulation of the two or more inputimages. In addition, the image processing unit 22 calculates theparallax between the two or more input images by using the modulationinformation and the superposed image and calculates the distance to anobject in the two or more input images on the basis of the calculatedparallax.

2. Detailed Configuration of Imaging System

Hereinafter, the imaging system 1 configured as described above will bedescribed in further detail.

FIG. 2 is a configuration diagram illustrating a specific configurationof the imaging system according to the first embodiment.

2-1. Imaging Apparatus

The imaging apparatus 10 includes an optical unit 100 and an imagingunit 110.

The optical unit 100 includes the two imaging optical systems 11L and11R and the two modulation masks 12L and 12R described above. Here, theimaging optical system 11L includes an optical system 101L constitutedby a lens or the like, a relay lens 103L, and a portion of a reflectiveoptical system 104 constituted by a prism or the like (portion on theleft side in FIG. 2). The modulation mask 12L is placed between theoptical system 101L and the relay lens 103L. For example, the modulationmask 12L is placed on an imaging plane of the optical system 101L. In asimilar manner, the imaging optical system 11R includes an opticalsystem 101R constituted by a lens or the like, a relay lens 103R, and aportion of the reflective optical system 104 (portion on the right sidein FIG. 2). The modulation mask 12R is placed between the optical system101R and the relay lens 103R. For example, the modulation mask 12R isplaced on an imaging plane of the optical system 101R.

The modulation masks 12L and 12R have mutually different opticaltransmittance distribution characteristics. In other words, each of themodulation masks 12L and 12R is a mask in which the opticaltransmittance differs at different positions therein, and thecorrelation coefficient of the optical transmittance distributioncharacteristics of the modulation masks 12L and 12R is less than 1. Forexample, the correlation coefficient of the optical transmittancedistribution characteristics of the modulation masks 12L and 12R issubstantially 0. In other words, the modulation masks 12L and 12R areuncorrelated. In addition, the optical transmittances at respectivesites in each of the modulation masks 12L and 12R are in uniformlyrandom numbers. However, the present disclosure is not limited to theuniformly random numbers, and the optical transmittances may be inGaussian random numbers.

The imaging unit 110 includes the imaging device 13 described above andan image output unit 112.

The imaging device 13 is a device that converts a two-dimensionaldistribution of optical intensities to electronic image data andcaptures a superposed image composed of two input images that have beenformed, modulated, and optically superposed on each other, as describedabove. The image output unit 112 outputs the superposed image capturedby the imaging device 13 to the image processing apparatus 20.

Although the modulation masks 12L and 12R are placed on the imagingplanes of the optical systems 101L and 101R, respectively, in theexample illustrated in FIG. 2, the modulation masks 12L and 12R mayinstead be placed at pupil positions of the optical systems 101L and101R, respectively.

FIG. 3 illustrates another example of the configuration of the opticalunit 100.

As illustrated in FIG. 3, the modulation masks 12L and 12R are placed aspupil positions of the optical systems 101L and 101R, respectively. Itis to be noted that the modulation masks 12L and 12R may instead beplaced at positions other than the positions of the imaging planes orthe pupils.

2-2. Image Processing Apparatus

The image processing apparatus 20 includes the acquiring unit 21 and theimage processing unit 22 described above.

The image processing unit 22 includes a modulation information storingunit 121, an image generating unit 122, a distance calculating unit 123,and a distance output unit 124.

The modulation information storing unit 121 stores information thatindicates a mode of modulation by each of the modulation masks 12L and12R, or in other words, stores modulation information, which is theinformation pertaining to the transmittances of these masks.

The image generating unit 122 generates reconstructed imagescorresponding to the two input images from mutually different viewpointsby splitting the superposed image acquired by the acquiring unit 21 withthe use of the modulation information stored in the modulationinformation storing unit 121.

The distance calculating unit 123 calculates the parallax between thetwo reconstructed images generated through the aforementioned split andcalculates the distance to an object in the reconstructed images on thebasis of the calculated parallax.

The distance output unit 124 outputs, as distance information, theinformation that indicates the distance calculated by the distancecalculating unit 123.

3. Processing Operation

With the configuration described above, in the imaging apparatus 10, aninput image I₁ and an input image I₂ obtained by the optical systems101L and 101R, respectively, are coded by the modulation mask 12L andthe modulation mask 12R that differ mutually, or in other words, aresubjected to luminance modulation. The imaging device 13 captures animage in which the input image I₁ subjected to first luminancemodulation by the modulation mask 12L and the input image I₂ subjectedto second luminance modulation by the modulation mask 12R are opticallysuperposed on and added to each other as an observation image y, whichis the superposed image described above. The image output unit 112 readsout the observation image y from the imaging device 13 and outputs theobservation image y.

3-1. Input Images and Observation Image

Now, the relationship between input images and an observation image willbe described in each of the case in which the modulation masks 12L and12R are placed on the imaging planes (the case of the arrangementillustrated in FIG. 2) and the case in which the modulation masks 12Land 12R are placed at the pupil positions (the case of the arrangementillustrated in FIG. 3).

When Modulation Masks 12L and 12R are Placed on Imaging Plane

When the modulation masks 12L and 12R are placed on the imaging planes,the coding indicated in the following expression (1) is carried out inthe imaging apparatus 10, in which the luminance values of the inputimages I₁ and I₂ are integrated with the transmittances of themodulation masks 12L and 12R, respectively, and the results are thenadded together.

y=A ₁ I ₁ +A ₂ I ₂  (1)

In the expression (1), y is the observation image. I₁ and I₂ are theluminance values of the respective input images and are each a vectorwith the number of elements N when the number of the pixels in the inputimage is N. A₁ is the optical transmittance of the modulation mask 12Land is a square matrix of N by N having, as a diagonal component, thetransmittance at each site of the modulation mask 12L corresponding toeach pixel position of the input image I₁. In a similar manner, A₂ isthe optical transmittance of the modulation mask 12R and is a squarematrix of N by N having, as a diagonal component, the transmittance ateach site of the modulation mask 12R corresponding to each pixelposition of the input image I₂.

When Modulation Masks 12L and 12R are Placed at Pupil Positions

When the modulation masks 12L and 12R are placed at the pupil positions,the imaging device 13 captures an observation image y as indicated inthe following expression (2). This observation image y is an image inwhich an image obtained by subjecting the input image I₁ to modulationdetermined by a first random pattern and an image obtained by subjectingthe input image I₂ to modulation determined by a second random patternare added to each other.

y=A′ ₁ I ₁ +A′ ₂ I ₂  (2)

The modulations A′₁ and A′₂ in the expression (2) will be described,hereinafter.

FIG. 4 illustrates an ideal optical system. As illustrated in FIG. 4, inthe ideal optical system, a divergent spherical wave emitted from apoint light source 301 on an object plane ξη enters an entrance pupil302 and then exits through an exit pupil 303 as a convergent sphericalwave, which forms an image 304 on an imaging plane ρν. Here, thedistance between the object plane ξη and the entrance pupil 302 is thedistance z₀, and the distance between the exit pupil 303 and the imagingplane μν is the distance z₁.

In reality, the image 304 is not imaged at one point on the imagingplane μν due to an influence of diffraction by the pupils and results inan image with a spread. This spread is referred to as a point spreadfunction and is represented by a Fraunhofer diffraction image of thepupil function indicated in the following expression (3).

$\begin{matrix}{{h\left( {u,v} \right)} = {\frac{Ca}{\lambda \; z_{i}}\underset{- \infty}{\overset{\infty}{\int\int}}{P\left( {x,y} \right)}{\exp \left( {{- j}\; \frac{2\pi}{\lambda \; z_{i}}\left( {{ux} + {vy}} \right)} \right)}{dxdy}}} & (3)\end{matrix}$

In the expression (3), Ca is a constant, and λ is the wavelength oflight. P(x,y) is the pupil function and matches the spatial distributionof the transmittances of the modulation masks 12L and 12R when themodulation masks 12L and 12R are placed at the pupil positions.

It is to be noted that the diagonal components of the transmittances A₁and A₂ of the modulation masks 12L and 12R, respectively, in theexpression (1) take the values obtained by integrating the spatialdistribution P(x,y) of the above transmittances by an area correspondingto each pixel of the input image.

The optical transfer function (OTF) of an incoherent imaging system isdefined as in the following expression (4) through the point spreadfunction.

$\begin{matrix}{{{OTF}\left( {f_{x},f_{y}} \right)} = \frac{\int{\int_{- \infty}^{\infty}{{{h\left( {u,v} \right)}}^{2}{\exp \left( {{- j}\; 2\pi \; \left( {{f_{x}u} + {f_{y}v}} \right)} \right)}{dudv}}}}{\int{\int_{- \infty}^{\infty}{{{h\left( {u,v} \right)}}^{2}{dudv}}}}} & (4)\end{matrix}$

Obtained by discretizing the OTF of each of the spatial patterns of themodulation masks 12L and 12R and by expressing the results in matricesare Q₁ and Q₂, and a Fourier transform matrix and an inverse Fouriertransform matrix are F and F⁻¹, respectively. In this case, the OTFexpresses the optical transfer in the frequency domain, and thus thesampling in the real space of the above expression (2) is expressed bythe following expression (5).

y=F ₁ ⁻¹ Q ₁ FI ₁ +F ₂ ⁻¹ Q ₂ FI ₂  (5)

On the basis of the expression (2) and the expression (5) above, themodulations A′₁ and A′₂ can be expressed as in the following expression(6).

A′ ₁ =F ₁ ⁻¹ Q ₁ F A′ ₂ =F ₂ ⁻¹ Q ₂ F  (6)

When the modulation masks 12L and 12R are located at positions betweenthe imaging planes and the pupils, the modulations of the input imagesI₁ and I₂ are the modulations in which the diffraction effect by theexit pupils (circular pupil in the case illustrated in FIG. 4) and thediffraction effect by the modulation masks 12L and 12R are combinedtogether.

On the basis of the above, regardless of the positions of the modulationmasks 12L and 12R, in either case of the expression (1) and theexpression (2) above, the observation image y is expressed as in thefollowing expression (7) as a linear transformation of an input image(I₁ ^(T), I₂ ^(T))^(T).

$\begin{matrix}{y = {\begin{pmatrix}A_{1} & A_{2}\end{pmatrix}\begin{pmatrix}I_{1} \\I_{2}\end{pmatrix}}} & (7)\end{matrix}$

In the expression (7) above, A is an observation matrix, and theobservation matrix A can be obtained in advance from the arrangement andthe transmittance distribution characteristics of the modulation masks12L and 12R in the optical unit 100. This observation matrix A is anexample of the modulation information described above.

3-2. Image Generation and Distance Calculation

The modulation information storing unit 121 stores the modulationinformation A, which is the information on the modulation masks 12L and12R obtained in advance. The image generating unit 122 obtainsreconstructed images I₁′ and I₂′ corresponding to the input images I₁and I₂ of the above expression (7) by using the modulation information Aand the observation image y captured by the imaging unit 110. In otherwords, the image generating unit 122 generates the two reconstructedimages I₁′ and I₂′ by splitting the observation image y.

$\begin{matrix}{y = {\begin{pmatrix}A_{1} & A_{2}\end{pmatrix}\begin{pmatrix}I_{1}^{\prime} \\I_{2}^{\prime}\end{pmatrix}}} & \left( 7^{\prime} \right)\end{matrix}$

Hereinafter, the method by which the image generating unit 122 obtainsthe reconstructed images I₁′ and I₂′ from the modulation information Aand the observation image y will be described. In the expression 7′, thenumber of variables in the reconstructed images I₁′ and I₂′ is 2N,whereas the number of expressions is N, which is smaller than the numberof variables. Thus, the solution is not determined uniquely with theexpression (7′) alone. Therefore, the reconstructed images I₁′ and I₂′are obtained from the modulation information A and the observation imagey through the expression (8) in which the sparsity is added as aconstraint condition. The sparsity is a property in which thereconstructed images I₁′ and I₂′ become sparse in a specific space suchas a frequency space. In other words, the sparsity is a property inwhich, when I₁′ and I₂′ are converted to the frequency space, a tinyportion of the coefficients becomes non-zero and the remainingcoefficients become zero or small enough values that can be regarded aszero. The problem of obtaining the reconstructed images I₁′ and I₂′ inthe expression (8) can be solved through the convex optimization.Specifically, the image generating unit 122 obtains c from the followingexpression (8) through a known convex optimization algorithm. The imagegenerating unit 122 then obtains the solution (I₁ ^(T), I₂^(T))^(T)=W⁻¹c from the obtained c. W is an operator that converts thesolution to a specific space and, for example, is a discrete cosinetransform (DCT).

$\begin{matrix}\left. \begin{matrix}{\arg \; {\min\limits_{c}\left( {{{y - {\begin{pmatrix}A_{1} & A_{2}\end{pmatrix}W^{- 1}c}}}_{2}^{2} + {\lambda {c}_{1}}} \right)}} \\{\begin{pmatrix}I_{1}^{\prime} \\I_{2}^{\prime}\end{pmatrix} = {W^{- 1}c}}\end{matrix} \right\rbrack & (8)\end{matrix}$

In the expression (8), λ is a weighting factor. Although the operator Win the expression (8) is, for example, a DCT, the operator W is notlimited to a DCT and may be any of a variety of transforms such as acurvelet transform or a wavelet transform or a combination thereof. Theconstraint condition in the expression (8) is not limited to the L1-normindicated above, and an L0-norm, an L2-norm, an L∞-norm, or an Lp-normin which p has a decimal number may instead be used. In addition, aregularization term aside from these norms may be used as the constraintcondition, or a plurality of combinations of these norms and theregularization term may also be used.

Through the processing described above, the image generating unit 122generates the reconstructed images I₁′ and I₂′ corresponding to theinput images I₁ and I₂ in the expression (7).

The distance calculating unit 123 acquires the reconstructed image I₁′and the reconstructed image I₂′ generated by the image generating unit122 and calculates the distance from the imaging apparatus 10 to theobject in each of the reconstructed image I₁′ and the reconstructedimage I₂′ through the distance calculation method based on the binocularstereoscopic vision. In other words, the distance calculating unit 123calculates the three-dimensional position including the distance. Thedistance output unit 124 outputs the three-dimensional positioncalculated by the distance calculating unit 123.

4. Simulation Result

FIGS. 5 through 7 illustrate examples of a simulation result accordingto the first embodiment. Specifically, FIGS. 5 through 7 illustrateexamples of a simulation result of processing in which two input imagesfrom different viewpoints obtained by the optical unit 100 of theimaging system 1 are modulated and superposed on each other and thedistance is calculated from the superposed image obtained through suchsuperposition.

FIG. 5 illustrates an exemplary arrangement and an exemplary imagingcondition of the imaging apparatus 10. As illustrated in (a) of FIG. 5,the imaging apparatus 10 is installed on the rear side of a vehicle.Then, as illustrated in (b) of FIG. 5, the imaging apparatus 10 capturesan image while the vehicle parks in a parking lot.

FIG. 6 illustrates input images of computer graphics, an observationimage based on these input images, and reconstructed images generatedfrom the observation image.

Specifically, the images illustrated in (a) and (b) of FIG. 6 are inputimages generated through computer graphics from images obtained by thetwo optical systems 101L and 101R in the imaging condition illustratedin (b) of FIG. 5. The image illustrated in (c) of FIG. 6 is an exampleof an image obtained by modulating the two images illustrated in (a) and(b) of FIG. 6 by the modulation masks 12L and 12R placed as illustratedin FIG. 2 and superposing the modulated images on each other, or inother words, is an example of an observation image. In this example,multi-valued uniformly random numbers are used as the opticaltransmittances at respective sites in each of the modulation masks 12Rand 12L. The images illustrates in (d) and (e) of FIG. 6 are examples ofthe reconstructed images I₁′ and I₂′ generated by the image generatingunit 122 with the use of the expression (8). This simulation resultreveals that the observation image can be split into the tworeconstructed images of the respective viewpoints.

FIG. 7 illustrates the distance calculated by the distance calculatingunit 123 and the true value of the distance for comparison. Thegradation in the images illustrated in (a) and (b) of FIG. 7 representsthe distance from the imaging apparatus 10. As illustrated in (a) ofFIG. 7, the distance calculating unit 123 calculates the distance fromthe imaging apparatus 10 to the object in each of the reconstructedimages I₁′ and I₂′ on the basis of the generated two reconstructedimages I₁′ and I₂′. The distance calculated by the distance calculatingunit 123 is close to the true value illustrated in (b) of FIG. 7.Accordingly, the imaging system 1 according to the present embodimentcan appropriately calculate the distance to the object.

5. Advantageous Effects

With the configuration described thus far, in the first embodiment, twoinput images from mutually different viewpoints are superposed on eachother on the single imaging device 13, the superposed image obtained bythe imaging of the imaging device 13 is split, and thus reconstructedimages corresponding to the original two input images can be generated.In other words, in order to obtain two input images, a conventionalimaging apparatus requires imaging devices corresponding to two cameras,but the imaging apparatus 10 according to the first embodiment requiresonly the imaging device 13 corresponding to a single camera.Accordingly, the circuit size of the imaging unit 110 can be kept small,and the cost can also be reduced advantageously.

In addition, in a conventional technique, data transmitted from animaging apparatus to an image processing apparatus that calculates thedistance needs to be in the amount corresponding to images in a numberequal to the number of viewpoints, for example, two images. In contrast,in the first embodiment, the amount of data of a superposed image (orobservation image) transmitted from the imaging apparatus 10 to theimage processing apparatus 20 can be reduced to the amount correspondingto a single image. In other words, the data transmission amount can bereduced.

In the first embodiment, the optical unit 100 includes the two imagingoptical systems 11L and 11R, and the image generating unit 122 generatestwo reconstructed images. The number of the imaging optical systemsincluded in the optical unit 100, however, is not limited to two and maybe three or more. In this case, the input images in the expression (7)are replaced with a matrix constituted by arranging input images I₁, I₂,I₃, . . . in a number equal to the number of the imaging opticalsystems. With this configuration, the reconstructed images can begenerated in a number equal to the number of the imaging opticalsystems, or in other words, equal to the number of the viewpoints, andthe distance can be calculated accordingly with the operation identicalto that of the first embodiment.

Modification of First Embodiment

In the first embodiment described above, a plurality of input images aresuperposed on each other on the single imaging device 13, and thus theimaging optical systems 11L and 11R of the optical unit 100 include therelay lenses 103L and 103R and the reflective optical system 104constituted by a prism or the like, as illustrated in FIGS. 2 and 3. Incontrast, constituent elements included in the imaging optical systems11L and 11R of the optical unit 100 according to the present disclosureare not limited to the relay lenses 103L and 103R and the reflectiveoptical system 104. Imaging optical systems 11L and 11R of an imagingapparatus 10 according to the present modification have configurationsdifferent from those illustrated in FIGS. 2 and 3.

FIG. 8 illustrates a configuration of the imaging optical systems 11Land 11R of the optical unit 100 according to the present modification.

For example, as illustrated in FIG. 8, the imaging optical systems 11Land 11R of the optical unit 100 according to the present modificationinclude decentered optical systems 105L and 105R, in place of the relaylenses 103L and 103R and the reflective optical system 104. With such aconfiguration as well, the imaging optical system 11L forms an inputimage modulated by the modulation mask 12L corresponding to the imagingoptical system 11L onto the imaging device 13. In a similar manner, theimaging optical system 11R forms an input image modulated by themodulation mask 12R corresponding to the imaging optical system 11R ontothe imaging device 13.

FIG. 9 illustrates another configuration of the imaging optical systems11L and 11R of the optical unit 100 according to the presentmodification.

For example, as illustrated in FIG. 9, the imaging optical systems 11Land 11R of the optical unit 100 according to the present modificationinclude freeform mirror optical systems 106L and 106R, in place of therelay lenses 103L and 103R and the reflective optical system 104. Withsuch a configuration as well, the imaging optical system 11L forms aninput image modulated by the modulation mask 12L corresponding to theimaging optical system 11L onto the imaging device 13. In a similarmanner, the imaging optical system 11R forms an input image modulated bythe modulation mask 12R corresponding to the imaging optical system 11Ronto the imaging device 13.

In other words, in either of the configurations illustrated in FIGS. 8and 9, an image obtained by modulating input images from the respectiveviewpoints of the optical systems 101L and 101R with the modulationmasks 12L and 12R and by then superposing the modulated input images oneach other can be captured by the imaging device 13 as a superposedimage or an observation image.

Accordingly, as in the first embodiment described above, the imagegenerating unit 122 can generate the input images I₁ and I₂ obtained bythe optical systems 101L and 101R, respectively. Furthermore, thedistance calculating unit 123 can calculate the distance to the objectin the input images I₁ and I₂.

When the reflective optical system 104 is used, as in the firstembodiment described above, attenuation of light in the prism isrelatively large. In the present modification, however, the decenteredoptical systems 105L and 105R or the freeform mirror optical systems106L and 106R are used, and thus attenuation of light in the opticalsystems can advantageously be kept sufficiently small. In other words,an image that is bright and with less noise can advantageously beobtained by the imaging device 13.

Second Embodiment

In the first embodiment, reconstructed images corresponding to inputimages from two viewpoints are generated from a superposed imageobtained by superposing input images from two viewpoints on each other,and the distance is then calculated on the basis of the binocularstereoscopic vision. In contrast, in a second embodiment, from asuperposed image obtained by modulating input images from two viewpointsand superposing the modulated input images on each other, reconstructedimages corresponding to the input images from the two viewpoints aregenerated and, at the same time, the parallax between these tworeconstructed images is estimated. With this configuration, the accuracyin calculating the distance to the object can be improved.

Hereinafter, the second embodiment will be described in detail.

An imaging system 1 according to the second embodiment has the sameconfiguration as that of the first embodiment. In other words, theimaging system 1 has the configurations illustrated in FIGS. 1 through3. The imaging system 1 according to the second embodiment differs,however, from the imaging system 1 according to the first embodiment interms of the operation of the image generating unit 122 and the distancecalculating unit 123. Specifically, in the second embodiment, the imagegenerating unit 122 generates two reconstructed images I₁′ and I₂′ froman image captured by the imaging apparatus 10, or in other words, from asuperposed image obtained by superposing input images from twoviewpoints on each other. At this point, the image generating unit 122also calculates the parallax between the two reconstructed images I₁′and I₂′ concurrently. Then, the distance calculating unit 123 calculatesthe distance to the object on the basis of the parallax between the tworeconstructed images I₁′ and I₂′ calculated by the image generating unit122 and parameters of the two optical systems 101L and 101R. Theparameters of the two optical systems 101L and 101R are parametersnecessary for binocular stereoscopic vision and include, for example,the distance between the optical systems 101L and 101R.

1. Image Generating Unit

Hereinafter, the operation of the image generating unit 122 according tothe second embodiment will be described in detail.

The image generating unit 122 acquires an observation image y from theimage output unit 112 via the acquiring unit 21. The image generatingunit 122 then obtains reconstructed images I₁′ and I₂′ from theobservation image y. The reconstructed images I₁′ and I₂′ correspond toinput images I₁ and I₂ obtained by the optical systems 101L and 101R,respectively. When obtaining the reconstructed images I₁′ and I₂′, theimage generating unit 122 uses the evaluation formula indicated in theexpression (8) in the first embodiment. In the second embodiment,however, the image generating unit 122 uses a different evaluationformula and thus estimates the reconstructed images I₁′ and I₂′ and theparallax between the reconstructed images I₁′ and I₂′ at the same time.Specifically, the image generating unit 122 uses the followingexpression (9) as the aforementioned different evaluation formula. Inother words, the image generating unit 122 obtains c and D that satisfythe expression (9) through a convex optimization algorithm and obtainsthe reconstructed image (I₁ ^(T), I₂ ^(T))^(T)=W⁻¹c from c.

$\begin{matrix}\left. \begin{matrix}{\arg \; {\min\limits_{c,D}\left( {{{y - {\begin{pmatrix}A_{1} & A_{2}\end{pmatrix}W^{- 1}c}}}_{2}^{2} + {\lambda_{1}{c}_{1}} + {\lambda_{2}{{{{Ms}\left( {- {DI}} \right)}W^{- 1}c}}_{1}}} \right)}} \\{\begin{pmatrix}I_{1}^{\prime} \\I_{2}^{\prime}\end{pmatrix} = {W^{- 1}c}}\end{matrix} \right\rbrack & (9)\end{matrix}$

In the expression (9) above, λ₁ and λ₂ are weighting factors, and Ms andD are as follows.

$\begin{matrix}{\mspace{20mu} {{{Ms} = {{\begin{bmatrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{bmatrix}.D} = \begin{bmatrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{bmatrix}}},{\quad{\begin{bmatrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{bmatrix},\ldots \mspace{14mu},{\begin{bmatrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{bmatrix}.}}}}} & (10)\end{matrix}$

The third term in the expression (9) is a parallax constraint term andis a constraint that the translated reconstructed image I₁′ matches thereconstructed image I₂′ with respect to the generated two reconstructedimages I₁′ and I₂′. Ms is a mask matrix for extracting a partial regionof an image and is a square matrix of N by N. The parallax D is aparallax matrix for translating the reconstructed image I₁′ by apredetermined parallax and is a square matrix of N by N. I is a unitmatrix of N by N.

In the expression (9), when it can be assumed that the parallax within asmall region of O pixels in an image of N pixels (N>O) is the same, Odiagonal elements of N diagonal elements in the mask matrix Ms are 1,and the other diagonal elements are 0. For example, the mask matrix Msof the expression (9) is a matrix in a case in which it is assumed thatthe parallax within a small region constituted by four neighboringpixels is the same and includes four diagonal elements having a valueof 1. Then, the image generating unit 122 repeats the processing ofcalculating c and D that satisfy the expression (9) for each smallregion of O pixels and thus calculates all the pixels of thereconstructed images I₁′ and I₂′ and the parallax D. It is to be notedthat the diagonal elements described above are also referred to asdiagonal components.

In this manner, in the second embodiment, when generating tworeconstructed images, the image generating unit 122 calculates the tworeconstructed images as well as the parallax between the tworeconstructed images on the basis of an evaluation value. The evaluationvalue, for example, is the value indicated by the function followingargmin in the expression (9) above. Specifically, when generating thetwo reconstructed images, the image generating unit 122 calculates thevalue of a first parameter corresponding to the two reconstructed imagesand the value of a second parameter corresponding to the parallaxbetween the two reconstructed images that minimize the evaluation valuethat is based on the first parameter and the second parameter. Then, theimage generating unit 122 converts the calculated value of the firstparameter to the two reconstructed images and acquires the calculatedvalue of the second parameter as the parallax. The first parameter, forexample, is c in the expression (9) above, and the second parameter, forexample, is D in the expression (9) above.

To be more specific, the evaluation value is a sum of the valuesindicated by a first term, a second term, and a third term. The firstterm indicates, by using the first parameter, the sum of squares of adifference between the superposed image and the images obtained bymodulating the two reconstructed images with the use of the modulationinformation. The second term indicates a value obtained by weighting anL1-norm of the first parameter. The third term indicates, by using thefirst parameter and the second parameter, a value obtained by weightingan L1-norm of a difference between an image obtained by translating oneof the two reconstructed images by a distance corresponding to theparallax and the other one of the two reconstructed images.

With this configuration, in the second embodiment, from a superposedimage obtained by superposing input images from two viewpoints,reconstructed images corresponding to the input images from the twoviewpoints and the parallax between these two reconstructed images canbe calculated at the same time. Furthermore, the accuracy in calculatingthe parallax can be improved.

2. Distance Calculating Unit

The distance calculating unit 123 calculates the three-dimensionalposition including the distance to the object in the reconstructedimages I₁′ and I₂′ from the parallax D calculated by the imagegenerating unit 122.

3. Simulation Result

FIGS. 10 and 11 illustrate examples of a simulation result according tothe second embodiment. Specifically, FIGS. 10 and 11 illustrate examplesof a simulation result of processing in which two input images fromdifferent viewpoints obtained by the optical unit 100 of the imagingsystem 1 are superposed on each other and the distance is calculatedfrom the superposed image obtained through such superposition.

The images illustrated in (a) and (b) of FIG. 10 are examples of theinput images obtained by the two optical systems 101L and 101R,respectively. Since this is a simulation, these images are standardimages, and each image is composed of 368×288 pixels=105984 pixels. Theimages illustrated in (c) and (d) of FIG. 10 are images of themodulation masks 12L and 12R, respectively. The image illustrated in (e)of FIG. 10 is an example of an observation image obtained by modulatingthe two input images illustrated in (a) and (b) of FIG. 10 with themodulation masks 12L and 12R illustrated in (c) and (d) of FIG. 10 andby superposing the modulated input images on each other. In thisexample, normally distributed random numbers are used for the opticaltransmittances at respective sites in each of the modulation masks 12Land 12R.

The images illustrated in (a) and (b) of FIG. 11 are examples of thereconstructed images I₁′ and I₂′, respectively, generated by the imagegenerating unit 122 with the use of the expression (9). Here, the imagegenerating unit 122 estimates the reconstructed images I₁′ and I₂′ andthe parallax therebetween for each rectangular region composed of 16×32pixels=512 pixels. This simulation result reveals that the imagegenerating unit 122 can split the observation image illustrated in (e)of FIG. 10, or in other words, the superposed image into the tworeconstructed images of different viewpoints. The image illustrated in(c) of FIG. 11 is an example of an image that indicates the parallax foreach rectangular region calculated by the image generating unit 122 withthe use of the expression (9). The image illustrated in (d) of FIG. 11is an image that indicates the correct parallax. The gradation of blackand white indicates the parallax in the images illustrated in (c) and(d) of FIG. 11. This simulation result reveals that the image generatingunit 122 can obtain the parallax that is substantially equal to thecorrect parallax.

4. Advantageous Effects

As described thus far, in the second embodiment, in a similar manner tothe first embodiment, two input images from mutually differentviewpoints are superposed on each other on the single imaging device 13,the superposed image obtained by the imaging of the imaging device 13 issplit, and thus the original two reconstructed images can be generated.Accordingly, in a similar manner to the first embodiment, the number ofthe imaging device 13 can be kept smaller than the number of theviewpoints, and the circuit size of the imaging unit 110 canadvantageously be reduced.

Furthermore, in the second embodiment, when generating the reconstructedimages I₁′ and I₂′, the image generating unit 122 uses the expression(9), which is an evaluation formula in which a parallax constraint thatthese two images are locally alike is added to the expression (8). Theuse of this evaluation formula to which such a parallax constraint isadded makes it possible to obtain the reconstructed images I₁′ and I₂′with less error, and the parallax D with less error can be calculated asa result. In other words, the accuracy in calculating the distance tothe object can be improved.

In the second embodiment as well, in a similar manner to the firstembodiment and the modification thereof, although the imaging apparatus10 includes the two imaging optical systems 11L and 11R and the twomodulation masks 12L and 12R, the number of the imaging optical systemsand the number of the modulation masks are each not limited to two andmerely need to be at least two. When the imaging apparatus 10 includestwo or more imaging optical systems and two or more modulation masks,the image processing apparatus 20 generates two or more reconstructedimages and calculates the parallax among these reconstructed images.Then, the image processing apparatus 20 calculates the distance to theobject in the two or more reconstructed images on the basis of thecalculated parallax.

Other Embodiments

Thus far, the imaging apparatus, the image processing apparatus, and theimaging system according to the present disclosure have been describedon the basis of the first embodiment, the modification thereof, and thesecond embodiment, but the present disclosure is not limited to theseembodiments and modifications thereof. Unless departing from the spiritof the present disclosure, an embodiment obtained by making variousmodifications that are conceivable by a person skilled in the art to theembodiments or an embodiment obtained by combining constituent elementsin different embodiments may also be included within the scope of thepresent disclosure.

For example, the image processing apparatus 20 according to the firstembodiment, the modification thereof, and the second embodimentdescribed above may be configured as a computer.

FIG. 12 is a block diagram illustrating a configuration of the imageprocessing apparatus 20 configured as a computer.

The image processing apparatus 20 includes an interface 305, a centralprocessing unit (CPU) 301, a read-only memory (ROM) 302, a random-accessmemory (RAM) 303, and a hard disk drive (HDD) 304. The interface 305 isa hardware piece that corresponds to the acquiring unit 21 and thedistance output unit 124. The CPU 301 has a function of the imagegenerating unit 122 and the distance calculating unit 123. The ROM 304stores, for example, a software program to be loaded and executed by theCPU 301. In other words, the CPU 301 loads and executes the softwareprogram stored in the ROM 304 and thus implements the function of theimage generating unit 122 and the distance calculating unit 123. The RAM303 temporarily stores, for example, data generated through theprocessing of the CPU 301. The HDD 304 serves as the modulationinformation storing unit 121 and stores modulation information.

The optical systems 101L and 101R in the first embodiment, themodification thereof, and the second embodiment described above merelyneed to be arranged such that the parallax is produced between aplurality of images of an object the distance to which is to be measuredand may be arranged horizontally or vertically.

In the first embodiment, the modification thereof, and the secondembodiment, the number of the imaging optical systems, the number of themodulation masks, and the number of the input images are each two, butsuch numbers are not limited to two and can be any number that is atleast two.

In the present disclosure, all or some of the units and the devices, orall or some of the functional blocks in the block diagrams illustratedin FIGS. 1 through 3 may be implemented by one or more electroniccircuits including a semiconductor device, a semiconductor integratedcircuit (IC), or a large scale integration (LSI). The LSI or the IC maybe integrated into a single chip or may be constituted by a combinationof a plurality of chips. For example, functional blocks other than amemory device may be integrated into a single chip. The name an LSI oran IC is used herein, but such a chip may also be called a system LSI, avery large scale integration (VLSI), or an ultra large scale integration(ULSI) depending on the degree of integration. A field-programmable gatearray (FPGA) that can be programmed after manufacturing an LSI or areconfigurable logic device that allows reconfiguration of theconnection within the LSI or setup of circuit cells within the LSI canalso be used for the same purpose.

Furthermore, all or some of the functions or the operations of theunits, the devices, and a portion of the devices can be implementedthrough software processing. In this case, the software is recorded onone or more non-transitory recording media such as a ROM, an opticaldisk, or a hard disk drive, and when the software is executed by aprocessor, the software causes the processor and the peripheral devicesto execute specific functions within the software. A system or anapparatus may include such one or more non-transitory recording media onwhich the software is recorded, a processor, and any necessary hardwaredevices such as an interface.

The image processing apparatus 20 in the first embodiment, themodification thereof, and the second embodiment described abovegenerates two or more reconstructed images by splitting a superposedimage but may calculate the distance to the object without generatingthe two or more reconstructed images.

In this case, the image processing apparatus 20 is configured as arange-finding apparatus and includes the acquiring unit 21 and the imageprocessing unit 22, as illustrated in FIG. 1. The acquiring unit 21acquires a superposed image composed of two or more input images frommutually different viewpoints that have been modulated and opticallysuperposed on each other. The image processing unit 22 calculates theparallax between the two or more reconstructed images by using themodulation information that indicates the mode of the modulation of thetwo or more input images and the superposed image and calculates thedistance to the object in the two or more reconstructed images on thebasis of the calculated parallax.

Although the imaging apparatus, the image processing apparatus, and theimaging system according to an aspect of the present disclosure havebeen described with the use of the first embodiment, the modificationthereof, and the second embodiment, the present disclosure may also bean imaging method and an image processing method to be implemented bythese apparatuses or systems.

FIG. 13A is a flowchart illustrating an imaging method according to anaspect of the present disclosure.

The imaging method according to an aspect of the present disclosureincludes step S11 and step S12.

Step S11

In this imaging method, first, with the use of the two or more imagingoptical systems and the two or more modulation masks, two or more inputimages from mutually different viewpoints are modulated and imaged onthe imaging device 13.

Step S12

Then, the imaging device 13 captures a superposed image composed of thetwo or more input images that have been formed, modulated, and opticallysuperposed on each other.

FIG. 13B is a flowchart illustrating an image processing methodaccording to an aspect of the present disclosure.

The image processing method according to an aspect of the presentdisclosure includes step S21 and step S22.

Step S21

In this image processing method, first, a superposed image composed oftwo or more input images from mutually different viewpoints that havebeen modulated and optically superposed on each other is acquired.

Step S22

Then, two or more reconstructed images are generated by splitting thesuperposed image with the use of modulation information that indicatesthe mode of modulation of the two or more input images.

FIG. 13C is a flowchart illustrating an image processing methodaccording to another aspect of the present disclosure.

The image processing method according to another aspect of the presentdisclosure includes step S31 and step S32.

Step S31

In this image processing method, first, a superposed image composed oftwo or more input images from mutually different viewpoints that havebeen modulated and optically superposed on each other is acquired.

Step S32

Then, the parallax between the two or more input images is calculated byusing modulation information that indicates the mode of modulation ofthe two or more input images and the superposed image and calculates thedistance to the object in the two or more input images on the basis ofthe calculated parallax.

In each of the embodiments described above, each constituent element maybe constituted by a dedicated hardware piece or may be implemented byexecuting a software program suitable for each constituent element. Eachconstituent element may be implemented as a program executing unit, suchas a CPU or a processor, loads and executes a software program recordedon a recording medium, such as a hard disk or a semiconductor memory.Here, the software program that implements the image processingapparatus and so on of the embodiments described above is a program thatcauses a computer to execute the steps in the flowcharts illustrated inFIG. 13B or 13C.

The imaging apparatus according to the present disclosure provides anadvantageous effect that the circuit size can be reduced and can beapplied, for example, to a rear-view monitor or the like to be mountedin a vehicle.

1-19. (canceled)
 20. An apparatus, comprising: an acquirer that acquiresa superposed image composed of two or more resulting images viewed frommutually different viewpoints, two or more input images being modulatedto generate the two or more resulting images, the two or more resultingimages being optically superposed to generate the superposed image; andan output that outputs information indicating a distance to an object,reconstructed images including images of the object, wherein thereconstructed images corresponding to the two or more input images aregenerated on the basis of the superposed image and modulationinformation that indicates modulation modes corresponding to the two ormore resulting images, wherein the modulation modes are different,wherein a parallax between the reconstructed images is calculated on thebasis of the superposed image and the modulation information, andwherein the distance is calculated on the basis of the parallax withoutusing the reconstructed images.
 21. A method, comprising: acquiring asuperposed image composed of two or more resulting images viewed frommutually different viewpoints, two or more input images being modulatedto generate the two or more resulting images, the two or more resultingimages being optically superposed to generate the superposed image; andoutputting information indicating a distance to an object, reconstructedimages including images of the object, wherein the reconstructed imagescorresponding to the two or more input images are generated on the basisof the superposed image and modulation information that indicatesmodulation modes corresponding to the two or more resulting images,wherein the modulation modes are different, wherein a parallax betweenthe reconstructed images is calculated on the basis of the superposedimage and the modulation information, and wherein the distance iscalculated on the basis of the parallax without using the reconstructedimages.